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ABSTRACT 

This study explores a technique for evaluating 
teacher education programs in terms of teaching competencies^ as 
applied to the Indiana University Mathematics Methods Program (HMP) . 
The evaluation procedures formulated for the study include a process 
product design in combination irith a modification of Pophan's 
performance test paradigm and Gage's adaptation of the microteaching 
concept. Ten preservice elementary education majors were randomly 
selected from the MMP and from the Mathematics Education Component of 
two alternative programs of teacher education (Contrast I and 
Contrast II) at Indiana University. Each preservice teacher (PST) was 
placed in a videotaped miniteaching situation where pupil learning 
was the criterion of effectiveness. Three teaching 

variables — Clarity^ Questioning^ and Involvement — were analyzed^ and 
PST mathematics knowledge relative to the teaching topic was 
assessed. The model yielded consistent end useful information in a 
short time span and in a relatively objfjctive manner. The three 
dimensions of teaching style correlate i^ighly with pupil achievement 
and MMP trainees achieved higher mean perforffiance scores than other 
PSTs on teaching style and pupil achievement variables. Finally^ 
pupils taught by MMP and Contrast II trainees achieved higher 
adjusted scores than those taught by the Contrast I groups which had 
no field experience in conjunction with their program of training. 
(DMT) 
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ABSTRACT 



This 



evaluating teacher education programs 



i: 



valuatl/e procedures involved a 



pre 



:h a modification of Popham's 



perf orn:- , 



pat ^ 



c thi mlcroteachlng concept, and 



were applied to the Indiqi:a 



Mathematics-Methods Program (MMP). 



Ten preservlce elementary educd-icn majors were randomly sampled from 
the MMP and from the mathematics education component of two alternative 
programs of teacher education at Indiana TIniversity. Each preservlce 
teacher (PST) was placed in a videotaped mini-teaching situation where 
pupil learning was the criterion of effectiveness. Three teacher variables 
were also analyzed and PST mathematics knowledge relative to the teaching 
topic was assessed. Significant differences between groups occurred in 
both the multivariate and subsequent univariate aiialyses. Significant 
correlations between the three teacher variables and pupil achievement 
were also found, encouraging further study In chese areas. 
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One vital and crucial element which should accomp. ^ the develop- 
ment of teacher education programs — and one which is o Ljcted, 
Is that of program assessment or evaluation. Elam (1971, speaking for 
the American Association of Colleges for Teacher Educatic cited evaluation 
attempts as one of the weakest elements xn teacher education. However, 
in reviewing the state of research related to teacher education, Peck 
and Tucker (1973) noted that both the quality of design and the report- 
ing of such research have improved since the mid-sixties. In particular, 
the use of observation instruments (category systems, rating scales) or 
some adaptation of the microteaching technique In recent research studies 
have helped educators to focus more clearly on and to delineate more 
carefully components of teacher/pupil behavior in the interactive setting. 
Further, an increasing number of process-product studies has been noted. 
These investigations, which relate teaching (process^ ehaviors to pupil 
outcome (product) measures, have been extensively revi vved by Rosenshine 
and Furst. In a 1973 AERA publication these educators reported nine 
variables which appear to have yielded the most significant and consistent 
correlations between aspects of teaching strategy or style and measures 
of student growth: clarity of presentation, variability, enthusiasm, 
task orientation, teacher indirectness, student opportunity to learn 
criterion materials teacher use of structuring comments, teacher use of 
multiple question types, and an absence of criticism. 

At the 1973 symposium of the American Educational Research Association, 
Popham described a teaching "performance test" paradigm which assesses 
teacher competency in terms of pupil achievement. In Popham's model a 
teacher is given a set of instructional objectives, prepares and teaches 
a mini-lesson to a small group of students in an effort to meet those 
objectives, and is subsequently evaluated in terms of the amount of 
learner growth he or she is able to effect. Popham's paradigm is based 
on the idea that a teacher education program is as good as the teachers 
it produces. Since his paradigm provides an assessment of teacher com- 
petency to meet prespecified objectives, the assessment simultaneously 
validates the program which set out to promote such teacher abilities. 

Popham's paradigm, however, concentrates only on product or outcome 
measures of teaching. Teacher educators also have a vested interest in 
process or teaching behaviors which are influenced by their program experi- 
ences and activities. Therefore, a slight modification of Popham's design, 
one which permits the study of specific teacher variables as well-, is 
desirable. The use of videotape to record the teaching performance test, 
for example, makes possible the analysis of teacher variables arid their 
relation to pupil learning without distorting Popham's basic paradigm. 
Gage (1968) has described this use of videotaping as an adaptation of the 
microteaching concept. Allen (1966;297), who coined the term, refers to 
microteaching as a mini-teaching encounter, one scaled down in terms of 
class size (1-5 pupils) and time (5-20 minutes). Elsewhere, Allen and 
Ryan (1969) state that microteaching is real teaching* The technique. 
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however, lessens the complexities of normal classroom teaching and makes 
possible the focusing upon specific teaching behaviors. Mlcroteachlng 
commonly Involves a teach-re teach cycle, where videotape feedback Is 
used to facilitate growth In teaching skills. In adapting the micro- 
teaching concept to process-product research, one can use the videotape 
feature to record the teaching session (s) for subsequent analysis of 
specific teaching behaviors. 

The purpose of the present study was to explore a technique for 
evaluating teacher education programs In terms of teaching competencies, 
and, In particular, to explore that technique as applied to the Indiana 
University Mathematics-Methods Program. The evaluation procedures formulated 
for the study Involve a t ^ocess-product design in combination with a 
modification of Popham's performance test paradigm and Gage's adaptation 
of the mlcroteachlng concept. Such a combination makes possible an in- 
depth analysis of specific teaching behaviors as well as an assessment 
of pupil learning which can be related to the teaching. 

More specifically the evaluation technique Involves: 

1. The identification of teacher variables skills and/or strategies) 

a. which research has shown to be conslsr.ently related to pupil 
cognitive/affective growth; and 

b. which are related to the teacher education program's philosophy 
or content materials. 

2. The utilization of Popham's performance test paradigm to place 
teachers from alternative programs in a mini-teaching situation 
(specific to the content field) where learner growth is the 
criterion of effectiveness. 

3. The videotaping of the teaching session (s) so that an in-depth 
analysis of the teacher variables identified in (1) can be facllltatea. 

In order to explore the effectiveness of the evaluation model, the 
procedures outlined above were used to assess the Indiana University 
Mathematics-Methods Program, an innovative program in mathematics education 
which has had nationwide Impact. The present study dealt only with the 
MM? component at Indiana University (1973-74 academic year), where the 
Program was Implemented in two pilot classes as a 12-hour, two-semester 
course. Program experiences offered elementary preservlce teachers an 
activity-oriented, field-based program with Integrated focus on both 
mathematics and relevant teaching methods. 

Three of the variables identified by Rosenshlne and Furst (1973) : 

(1) clarity of presentation, (2) teacher use of multiple question type*i 
and (3) teacher Indirectness, were chosen as the bases for analyzing the 
teaching styles and strategies of the preservlce elementary teachers 
participating in this investigation. A slight rephrasing of the above 
variable names was used, however, to emphasize the connotation given them 
in the study: (1) clarity in developing a mathematical idea (Clarity), 

(2) inquiry questioning techniques (Questioning), and (3) encouragement 
of constructive pupil Involvement (Involvement). 

Three factors motivated the choice of these dimensions of teaching 
style/strategy. First, research has shown them to be consistently related 
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to pupil cognitive growth. Second, of the variables Identified by 
Rosenshlne and Furst, tnese can be traced to elements In the philosophy 
and/or vnritten materials of the Hithematlcs-Methods Program (Dodd, 1974) « 
Third, it Is possible to code behaviors specific to these variables during 
a mini-teaching session such as that suggested by Popham's performance 
test. 

The study was designed to analyze theae three dimensions of the 
teaching style of prestarvlce teachers (PFTs) from the Mathematlcs- 
Methois Program and from the mathematics education component of two 
alternative teacher education programs at Indiana University. The 
mathematical knowled5/,e. of the preservice teachers relevant to the topic 
of the mini-teaching sessions was also assessed (PST Test). In addition 
the relationship betvf:»en these four teacher variables and mean adjusted 
pupil scores (?upil Score) was also investigated. 

Two major questions were asked: (1) Is there any difference in the 
performances of preservice teachers from the Mathematics-Methods Program 
and from the two alternative programs, on the multivariate composite or 
in the univariate: analyses, of the following dependent measures: Clarity, 
Questioning, Involvement, PST Test, Pupil Score? (2) Is there any re- 
lationship between the mean adjusted achievement scores of pupils taught 
by the PSTs and the other four dependent variables? 



METHOD 



Subjects 

The population of preservice teachers for the study consisted of 
undergraduate elementary education majors from the Mathematics-Methods 
Program (MMP) and from the mathematics education componetit of fwo al- 
ternative teacher education programs (Contrast I and Contrast T.I) at 
Indiana University. The Mathemiitics-Methods Program provides an in- 
tegrated program of mathematics nnd related teaching methods instruction 
in combination with regular elementary school experiences in teaching 
mathematical topics. Trainees in the other two programs had previously 
completed the required mathemaC 'cs courses for teachers ad were currently 
eniolled in a course of methods of teaching elementary mathematics. The 
Contrast I group had no field experiences in conjunction with their course. 
Or the other hand. Contrast II trainees regularly participated in school 
e}.periences. The field experiences of this latter group were, however, 
more general in nature than those of the IMP group, and ware not restricted 
to teaching mathematical topics t children. Ten preservice teachers were 
randomly sampled from each of the programs: MMP, Contrast I, and Contrast 
II. 

Third grade pupils from Rogers and University Elementary Schools in 
Bloomington, Indiana provided the school population. The sample con- 
sisted of 80 out of 92 University pupils and 40 of 66 Rogers School students 
The schools were chosen because of the similarity of the socio-economic 
background of pu; is and the extensive videotaping facilities at University 
School. A major portion of these pupils were children of professional 
businessmen or of Indiana University graduate students or faculty. Average 
I.Q. of pupils was 120. 
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Instrumentation 



Both written and observation Instruments were constructed by the 
Investigator for the study. The three written tests: Fraction Inventory 
(for school pupils). Pupil Equivalent Fraction Posttest, and PST 
Equivalent Fraction Test were each content validated by faculty members 
of the Indiana University Mathematics and Mathematics Education depart- 
ments, and piloted with several groups prior to the study. Reliability 
studies were carried out for each test and an was obtained for 

each test prior to Its use In the Investigation. 

Three observation Instruments and accompanying coders' guides were 
developed by the Investigator which speclfle<^ the characteristics of 
behavior that defined each of the three teacher variables Clarity, 
Questioning* and Involvement for the study. The major categories of each 
of these Instruments are listed below. 

Major Categories of the Three Observation Instruments 

1. Clarity In Developing a Mathematical Idea (rated) 

a. Mathematical objectives of the lesson are Identifiable 

b. Lesson Is well planned and executed 

c. Instruction builds upon previous learning and experiences 

d. Models and illustrations are effectively used 

e. Flow of ideas from Instructor to pupils is understood 

f . Evaluation is on-going, to check that ideas are clearly grasped 

2. Inquiry Questioning Tecr'.:lques 

a. General questioning ^oves of the preservlce teacher (rated) 

1. Used questioning rt-ther than exposition as basic strategy 

2. Sequences questions appropriately 

b. Inquiry questions asked by the preservlce teacher (counted) 

1. Systematic observation 

2. Rationalization 

3. Higher order extension of new learning 

3. Encouragement of Constructive Pupil Involvement 

a. Elicits student ideas which relate to important ideas in the 
lesson sequence (counted) 

b. Relates learning to child's world (counted) 

c. Builds upon student response (counted) 

d. Encourages activities which promote self-learning within the 
lessOn (counted) 

e. Structures lesson so pupils are actively participating in a 
manner which facilitates learning of the mathematical concepts 
relevant to the lesson. (rated) 



Four graduate students were trained as observers to view and code 
the videotaped teaching sessions of the PSTs who participated in the 
study. Videotapes from a pilot study were used to train coders. The 
training sessions focused on one variable at a time and allowed for 
open discussion until questions were answered and the Interpretation of 
behaviors for various categories of an instrument and accompanying 
coder's guide was agreed upon by all coders. When coders were confident 
of the coding procedures for each of the teacher variables, they in- 
dependently coded two of the training tapes. Each coder viewed each tape 
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twice. At one viewing the teacher's questioning style was observed and 
coded. At another viewing the coder concentrated upon the preservice 
teacher's encouragement of constructive pupil involvement. Having viewed 
a tape twice, the coder then assigned a clarity rating to the PST in- 
volved. Since the average of the scores assigned to a PST on a variable 
was recorded as the dependent measure, the procedure outlined by Snedecor 
(Ebel, 1967; 116-119) for average ratings was used to test inter-coder 
reliability. A criterion of .80 was set. The initial training sessions 
were considered complete when this criterion was met for each variable 
coded. The 30 videotapes were randomly assigned to coders in such a 
manner that each tape vas independently coded by two different persons. 
Both inter-coder and intra-coder reliability was calculated at periodic 
intervals throughout the time tapes were being analyzed. 



Procedures 

Prior to the teaching sessions each preservice teacher took the 
researcher-constructed PST Equivalent Fraction Test. The Kuhlmann- 
Anderson I.Q. Test (Form CD) and the Fraction Inventory were also 
administered as pretest measures to the third grade pupils of University 
and Rogers Elementary Schools. These latter two measures were used to 
adjust pupil posttest scores for initial differences in learning ability 
and knowledge of fractions. 

Popham's performance test paradigm provided the framework for the 
subsequent teaching sessions: 

1. Three days prior to teaching, each preservice teacher was given a 
list of instructional objectives for the two half -hour sessions he 

or she was to prepare and teach to a group of four third grade pupils. 
The topic was "equivalent fractions;" the lesson was introductory in 
nature. A summary of pupils' background work with fractions was also 
given to the PSTs. 

2. Each preservice teacher had three days to prepare his/her teaching 
sessions. He or she was asked to do so independently, rather than 
solicit peers or instructors for help. 

3. Each preservice teacher taught, on two consecutive days, his/her 
preplanned lesson to the group of 4 third graders randomly assigned 
to him/her. Each of the two half-hour sessions was recorded on 
videotape for subsequent analysis by coders. 

4. On the day following the second teaching session, students took the 
Pupil Equivalent Fraction Posttest. Adjusted group posttest scores 
were used as an estimate of the preservice teacher's effectiveness 
d ur ing ins tr uc t ion . 

By means of the written tests and analysis of the videotaped teach- 
ing sessions, the following five dependent measures were obtained for each 
preservice teacher: PST Equivalent Fraction Test score, clarity rating, 
que^' '^iing index, constructive involvement score, and mean adjusted pupil 
scoi The number of correct responses on each of the two written tests 
was calculated. For the PST Equivalent Fraction Test, this score was 
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used as the dependent measure. Further, for each preservice teacher, 
the adjusted group mean performance of his/her four pupils on the Pupil 
Equivalent Fraction Posttest was entered as a dependent measure. The 
average numeric sum of frequency counts and ratings recorded by the 
trained observers (using the three observation instruments and coders' 
guides) provided, respectively, a clarity rating, questioning indrx, 
and constructive involvement score for each preservice teacher. 



Satisfactory reliability information is necessary to support the 
credibility of data gathered from the instruments and to lend substance 
to interpretations or conclusions based on the analysis of that data. 
For this reason four reliability studies were carried out for the study, 
one for each of the researcher-constructed written instruments as well 
as a special study of coder consistency. The computer program TESTAT 
from the Indiana University Educational Statistics library was used for 
the analysis of the written tests. The alpha coefficient of internal 
consistency exceeded .83 for each of the tests. The Snedecor formula 
for average coder scores was used to determine both intra- and inter- 
coder reliability. The lowest reliability estimate obtained was .87 
(against the criterion .80), indicating the coders were generally con- 
sistent with themselves and with each other. 



To test differences between groups (MMP, Contrast I, Contrast II) 
on the five dependent variables, a completely randomized one-way multi- 
variate design with 10 subjects per cell was used. A subsequent univariate 
analysis was performed on each dependent variable. 

The mean scores and standard deviations of the five dependent variables 
for each of the three groups of preservice teachers are given in Table 1* 
Mean scores are presented in both raw and standardized form. 



RESULTS 



Reliability Studies 



Tests of Hypotheses Associated with v;roup Differences 



TABLE 1 



Means and Standard Deviations 
of the Three Groups of Preservice Teachers 



MMP 



Contrast I 



Contrast II 



Raw Score Means ; 
PST Test 
Clarity 



18.50 
32.65 
52.05 
59.00 
2.11 



16.30 
19.50 
29.65 
36.85 
-2.17 



16.70 
25.55 
41.25 
45.35 



Questioning 
Involvement 
Pupil Score 



.16 



EKLC 
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MMP 


Contrast I 


Contrast II 


Standard Score Means: 








PST Test 


.36 


- .23 


- .13 


CI n.l Ity 


.83 


- .87 


.05 


Q'lestlonlng 


.59 


- .61 


.01 


Involvement 


.79 


- .67 


- .11 


Pupil Score 


.82 


- .87 


.05 


Raw Score Standard 








Deviations: 








PST Test 


3.84 


3.71 


3.65 


Clarity 


9.15 


5.64 


5.16 


Questloiilng 


20.38 


10.57 


17.87 


Involvement 


16.01 


9.92 


10.48 


Pupil Score 


2.12 


1.61 


1.80 



*The mean adjusted pupil scores reported are residual gain scores. The 
Statistical Package for the Social Sciences computer program REGRESSION was 
used to compute the scores with I.Q. and Fraction Inventory scores as the 
predictor variables. 



Table 1 indicates that the rank order of mean performances on the 
five dependent variables of the study was constant. Preservice teachers 
from the MMP group performed highest, Contrast II PSTs ranked second, and 
PSTs from Contrast I performed lowest on all five variables. 

The multivariate F test reported in Table 2 indicated a significant 
variation among the mean vectors for groups with a p value less than .011. 
This table also presents univariate F statistics for each of the dependent 
variables. The univariate analyses are supported by Cramer and Bock 
(1966), who state that a significant multivariate statistic justifies an 
Investigation of separate univariate F tests for the corresponding variables. 
Significant differences among groups occurred for the variables Clarity, 
Questioning, Involvement, and Pupil Score. No significant differences were 
obtained among groups for the variable PST Test. 



TABLE 2 

Multivariate and Univariate Analyses 
of Variance for Hypotheses Testing Group Differences 



Dependent 



Variable (s) 


Test 


df 




F 


P< 


PST Test, Clarity, Questioning, 












Involvement, Pupil Score 


M 


10, 


46 


2.708 


.011 


PST Test 


U 


2, 


27 


.084 


.387 


Clarity 


U 


2, 


27 


9.144 


.001 


Questioning 


U 


2, 


27 


4.446 


.021 


Involvement 


u 


2, 


27 


8.062 


.002 



-8' 



Multiple comparisons between group means were also carried out for 
significant univariate F statistics. The Tukey procedure was chosen to 
decide which groups of preservice teachers were responsible for the 
significant F ratios since this method provided for pairwise comparisons 
of means for groups of equal size (Hopkins and Chadboum, 1967). This 
latter analysis indicated that PSTs from the Mathematics-Methods Program 
performed significantly higher than Contrast I on all four variables 
(p < .05), and higher than Contrast II on Clarity, Involvement, and Pupil 
Score (p < .10). In addition, PSTs from Contrast II (the group with a 
school experience component) performed significantly better on the Pupil 
Score variable than did Contrast I trainees (no school component). 



Tests of Hypotheses Which Investigated Correlation Between Variables 

Pearson product moment correlations were computed to determine the 
relationships of pupil achievement with the other four teacher variables. 
The correlations between Pupil Score and the variables Clarity, Question- 
ing, Involvement, and PST Test are presented in Table 3 . Three variables 
in this table (Clarity, Questioning, Involvement) are significant correlates 
of Pupil Score and, in addition, are significantly correlated with each 
other (p < .01) . 



TABLE 3 

Correlations Between Mean Adjusted Pupil Scores 
and the Other Dependent Variables Used in This Study 



Clarity 
Questioning 
Involvement 
Pupil Score 


.34* 
-.11 
.15 
.26 


.70** 
.79** 
.96** 


.83 

.71** .75** 




PST Test 


Clarity 


Questioning Invcilvement 



*p < .05 
*p < .01 



An extremely high correlation occurs between Clarity and Pupil Score (.96). 
The variable PST Test showed no significant correlation with the criterion, 
Pupil Score. 



DISCUSSION 



The present study explored a technique for evaluating teacher educa- 
tion programs in terms of the exhibited competencies of its trainees. 
Several conclusions, based upon the results of this exploration, may be 
stated. 
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1. The procedures used in this study should be considered a viable 
technique for evaluating segments of a teacher education program. 

2. The three teacher variables Clarity, Questioning, and Involvement are 
strong correlates of pupil achievement. 

3. The Mathematics^Methods Program appears to have promoted competencies 
associated with the variables Clarity, Questioning, Involvement, and 
Pupil Score more effectively than the other two programs of the study. 

4. There is an indication that regular, planned school experience in 
conjunction with the mathematics preparation of preservice elementary 
teachers may have an impact on teacher competency to produce mathe- 
matical learning in children. 

The evaluation model proposed and explored in this investigation 
yielded consistent and useful information for a specific teacher educa- 
tion program, and did so within a short time span and in a relatively 
objective manner. These factors encourage further use of the technique 
in the evaluation of other teacher education programs. The three dimensions 
of teaching style chosen for analysis in the present evaluation correlated 
highly with pupil achievement. These results parallel other findings in 
educational research (see Rosenshine, 1973). 

I«7hile MMP trainees achieved higher mean perforwmce scores than 
other PSTs on the teaching style and pupil achievement variables, a 
more detailed and rigorous analysis of components of the Mathematics- 
Methods Program is necessary before definitive conclusions comparing 
these programs can be drawn. Certainly present results would support 
such efforts. 

Conclusion 4 lends some support to the current trend **o include 
field experiences in teacher education programs. In the present study 
pupils taught by MMP and Contrast II trainees achieved higher adjusted 
scores than those taught by the Contrast I group, which had no school 
experiences in conjunction with their program of training. The school 
experiences of the MMP trainees were specific and followed a definite 
sequence laid out by their field instructors. Those of the Contrast II 
group, on the other hand, were more general in nature and were not 
limited to teaching mathematical topics. The mean adjusted pupil scores 
of MMP trainees were higher than those of Contrast II PSTs at the .10 
significance level. It would be interesting to explore these differences 
in the nature of field experiences more rigorously and systematically in 
order to assess the c "fectiveness of various types of such experiences 
on pupil achievement . 

Finally, several limitations of the generalizability of results are 
recognized. The evaluation model formulated in this study deals with only 
one phase of a program's evaluation: exhibited competency of trainees 
with respect to specific teacher variables and their ability to promote 
pupil learning. No attempt at total program assessment was made. An 
additional limitation is the fact that the sample of preservice teachers 
was not large. Also, the elementary school population was small and 
rather atypical. Further, all teaching sessions focused upon one topic 
(equivalent fractions) at one grade level (grade 3). This arrangement 
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did not allow for a study of teaching competencies with respect to 
other mathematical topics nor with children of different age levels. 
Each of these limiting factors implies a suggestion for further research. 
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